Moving vehicle detection and analysis using low resolution remote sensing imagery

ABSTRACT

A method comprises accessing a plurality of images of a substantially same geographical area, each image of the plurality of images captured at a separate time from each other using a separate imaging sensor, each imaging sensor capturing information corresponding to a different spectral band. The method further comprises identifying, for the images, a set of blobs, each blob comprising a plurality of adjacent pixels, wherein each image of the plurality of images includes a blob of the set of blobs, and wherein each blob in the set of blobs has a location in each image that differs from a location of other blobs in the set of blobs. The method further comprises generating a score indicating a likelihood that the set of blobs correspond to a moving object, storing an indication that the set of blobs correspond to the moving object responsive to the generated score exceeding a threshold.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. application Ser. No.15/351,373, filed Nov. 14, 2016 which claims benefit of U.S. ProvisionalApplication No. 62/256,072, filed Nov. 16, 2015, which is incorporatedby reference in its entirety.

BACKGROUND Field of Art

This disclosure relates generally to image processing of low resolutionimages and in particular to identifying moving vehicles in ageographical area using low resolution images captured by an aerialimaging device.

Description of Art

Several applications analyze aerial images to identify objects in theimages, for example, various objects in aerial images captured bysatellites. Analysis of high resolution images can be performed usingrelatively simple techniques. Obtaining high resolution aerial imagestypically requires use of large, expensive satellites and results. Thesesatellites typically require a significant amount of resources. Forexample, such satellites carry sophisticated and expensive equipmentsuch as high spatial resolution cameras, expensive transponders, andadvanced computers. Other factors that contribute to the cost associatedwith expensive imaging satellites are the launch cost and maintenance.Expensive high spatial resolution imaging satellites must be monitoredfrom a ground facility, which requires expensive manpower. Thesesatellites are also susceptible to damage or costly downtimes. The highlaunch and development costs of expensive imaging satellites leads to aslowdown in the introduction of new or upgraded satellite imagery andcommunication services for object detection.

Cheaper low spatial resolution imaging satellites and drones may be usedfor capturing images. However, such satellites and drones provideunclear images. In low-resolution imagery, objects such as vehicles aretypically not clearly identifiable and often appear as blobs containinga few adjacent pixels. In other instances, such as in infrared bandimagery, the images are completely invisible to humans. Conventionalimage processing techniques fail to identify objects such as movingvehicles, cars, trucks, or speedboats from low resolution images takenby cheaper low resolution imaging satellites.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed embodiments have advantages and features which will bemore readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

Figure (FIG. 1 illustrates a block diagram of an example systemenvironment in which a moving vehicle analysis system operates.

FIG. 2 illustrates a block diagram of an example system architecture forthe moving vehicle analysis system.

FIG. 3 illustrates an example low resolution aerial image taken frommultiple imaging sensors.

FIG. 4 illustrates an example process for a machine learning trainingengine in the moving vehicle analysis system.

FIG. 5 illustrates example training sets for a machine learning model inthe moving vehicle analysis system.

FIG. 6 illustrates an example process for the moving vehicle analysissystem for processing low resolution aerial imagery.

FIG. 7 illustrates an example application of the process for the movingvehicle analysis system to process low resolution aerial imagery.

FIG. 8 illustrates an example low resolution aerial image of ageographical area taken from multiple imaging sensors.

FIG. 9 illustrates an example low resolution aerial image containingpixel blobs.

FIG. 10 illustrates an example output of a traffic analysis module.

FIG. 11 illustrates example low resolution aerial images captured fromsingle imaging sensors at distinct times.

FIG. 12A illustrates an example output of the machine learning model fortwo moving vehicles.

FIG. 12B illustrates an example output of the traffic analysis modulefor two moving vehicles.

FIG. 13 is a block diagram illustrating components of an example machineable to read instructions from a machine-readable medium and executethem in a processor or controller.

DETAILED DESCRIPTION

The Figures (FIGS.) and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Configuration Overview

Disclosed by way of example embodiments are systems, methods and/orcomputer program products (e.g., a computer readable storage media thatstores instructions executable by one or more processing units) foridentifying moving vehicles from low resolution aerial imagery. In oneexample embodiment, a moving vehicle analysis system receives lowresolution images of a geographical area from an aerial imaging device,such as a satellite, drone, or other aerial configured imaging system.Within the aerial imaging device are multiple imaging sensors thatcapture images at different moments in time. A single imaging sensorcaptures an image at a particular time.

The moving vehicle analysis system extracts signals from noisy, lowresolution images and identifies moving vehicles in the images. Thepixel resolution of the received low resolution images is measured inGround Sample Distance (GSD) units. In one example, the resolution ofthe received images is in the range of 3-5 meters/pixel. At thisresolution, moving vehicles, such as cars, trucks, trains, ships, andspeedboats appear as ambiguous pixel blobs of a few adjacent pixelseach. Note that the terms “pixel blob” and “blob” may be usedinterchangeably in this description and each refers to a collection ofadjacent pixels approximately matching each other, based on a pixelattribute. For example, a blob may include five adjacent pixels whosebrightness attribute value is more than double that of all pixelssurrounding the blob. In another example, each pair of pixels within ablob may include pixels whose brightness attribute value is within athreshold of each other.

The moving vehicle analysis system may perform edge analysis in theimages to identify pairs of edges corresponding to vehicular paths. Avehicular path may be a path that a vehicle may travel upon or through,for example, a road, street, avenue, a shipping lane, railroad tracketc. The moving vehicle analysis system may identify pixel blobs in theimages by pixel clustering or other image processing techniques. Themoving vehicle analysis system may extract a feature vector from theimages. The feature vector may include feature describing the relativepositions of blobs with respect to each other and with respect tovehicular paths.

In other example embodiments, the moving vehicle analysis system mayprovide the feature vector as input to a machine learning model forgenerating a score indicative of a likelihood that the blobs correspondto moving vehicles. The moving vehicle analysis system may associateblobs with moving vehicles if the generated score exceeds a threshold.In one example embodiment, the moving vehicle analysis system maydetermine a count of the moving vehicles in the images, where eachmoving vehicle is associated with corresponding blobs. The movingvehicle analysis system may determine the size, speed, and direction ofeach moving vehicle. The moving vehicle analysis system may analyzetraffic patterns corresponding to the vehicular paths and sendre-routing information to the moving vehicles responsive to the analyzedtraffic pattern exceeding a threshold.

Example System Environment

Figure (FIG. 1 illustrates a block diagram of an example systemenvironment in which a moving vehicle analysis system 101 operates. Theexample system environment shown in FIG. 1 may include an aerial imagingdevice 110, a moving vehicle analysis system 101, a traffic database109, a client device 120, and a vehicle 130. The aerial imaging device110 may be a satellite, drone, or other aerial configured imagingsystem, capable of capturing low resolution images. It is noted that theimaging system within the aerial imaging device 110 may include multipleimaging sensors that capture at different moments in time. The aerialimaging device 110 may capture multiple images with a time lag. Themultiple images may correspond to the same spectral band or differentspectral bands, where a spectral band corresponds to a range ofwavelengths of light. Example spectral bands include the red spectralband, the green spectral band, the blue spectral band, the infraredspectral band, and the panchromatic spectral band.

A moving vehicle may appear as an ambiguous blob of adjacent pixels oneach image, as illustrated and described below in detail with referenceto FIG. 3. Because the images from the aerial imaging device 110 arecaptured at a time lag, moving vehicle blobs do not line up when theimages from multiple imaging sensors are combined. For example, in anaerial imaging device 110 where each imaging sensor corresponds to adifferent spectral band, moving vehicle blobs can appear at differentlocations in the red, blue, and green spectral bands. This becomesapparent when images from the different imaging sensors are combinedinto a color image. An example low resolution aerial image containingblobs combined from multiple imaging sensors is illustrated anddescribed below in FIG. 3.

The moving vehicle analysis system 101 illustrated in FIG. 1 may includean image store 102, an optional vehicular path detection module 105, anoptional pixel blob identification module 103, an optional featureextraction module 104, a machine learning model 106, a traffic analysismodule 107, and a bus 108. Optional components of the example systemenvironment illustrated in FIG. 1 are shown using dashed lines. Theimage store 102 shown in FIG. 1 may store low resolution images receivedfrom the aerial imaging device 110. The vehicular path detection module105, pixel blob identification module 103, feature extraction module104, and machine learning model 106 may retrieve the images stored inthe image store 102 for processing.

The optional vehicular path detection module 105 shown in FIG. 1performs edge analysis in the received images to identify pairs ofedges, where each pair of edges corresponds to a vehicular path, asillustrated and described below in detail with reference to FIG. 2. Aspreviously noted, the vehicular path may be a path along with a vehicletravels, for example, a road, street, avenue, highway, unpaved road,railroad track, etc. In one example embodiment, the pixel blobidentification module 103 shown in FIG. 1 takes images from the imagestore 102 and identifies blobs in each image. Each blob includesadjacent pixels that match each other based on a pixel attribute. In oneexample, a pixel attribute is a value from 0 to 256 that represents thepixel's brightness. Further details about how the moving vehicleanalysis system 101 may identify blobs based on pixel attributes aredisclosed in FIG. 2. The feature extraction module 104 may extract afeature vector from the received images describing the relativepositions of blobs with respect to the vehicular paths identified by thevehicular path detection module 105 and with respect to other blobs, asillustrated and described below in detail with reference to FIG. 2. Thefeature vector may be transmitted to the machine learning model 106 forgenerating a score, which is indicative of a likelihood that theidentified blobs correspond to one or more moving vehicles. Inalternative embodiments, the machine learning model 106 may generate,based on the received images, an output including pixel blobs (includingone or more adjacent pixels matching each other based on a pixelattribute) and a score indicative of a likelihood that the blobscorresponds to a moving vehicle.

The traffic analysis module 107 shown in FIG. 1 may store an associationbetween blobs and a moving vehicle if the generated score for the blobsexceeds a threshold. The traffic analysis module 107 may determine acount of one or more moving vehicles in the received images, where eachmoving vehicle is associated with corresponding blobs, as illustratedand described below in detail with reference to FIG. 2. The trafficanalysis module 107 also may determine speeds and sizes of movingvehicles. The traffic analysis module 107 also may analyze trafficpatterns corresponding to the time of capture of the images, the countof the moving vehicles, the size of each moving vehicle, and theiraverage speed. The traffic analysis module 107 also may send re-routinginformation to a client device 120 and vehicle 130 if the analyzedtraffic pattern exceeds a threshold such as a traffic congestion value.The image store 102, vehicular path detection module 105, pixel blobidentification module 103, feature extraction module 104, machinelearning model 106, and traffic analysis module 107 communicate by a bus108. Further details about the components of the moving vehicle analysissystem 101 are provided in FIG. 2.

The optional external traffic database 109 shown in FIG. 1 maycommunicate with the moving vehicle analysis system 101 for the machinelearning model 106 and traffic analysis module 107 to retrieve trafficinformation for processing. The traffic database 109 may containinformation about traffic volumes at intersections with highways(volume, speed, vehicle classification, truck weight data, etc.),traffic growth and trends, seasonal adjustment factors used indetermining estimates of annual average daily traffic (AADT) anddirectional design hour volume (DDHV) from the Department ofTransportation. The traffic database 109 may contain map data collectedfrom performing systematic ground surveys using tools such as a handheldGPS unit, a notebook, digital camera, voice recorder, etc., from theOpenStreetMap database. The machine learning model 106 may use theinformation in the traffic database 109 to identify blobs correspondingto moving vehicles. The traffic analysis module 107 may use theinformation in the traffic database 109 for analyzing traffic patternsincluding counts of moving vehicles, the size of each vehicle, theaverage speed, the density of vehicles, etc.

The moving vehicle analysis system 101 may interact with the clientdevice 120 shown in FIG. 1. The client device 120 may be a computingdevice capable of receiving client input as well as transmitting and/orreceiving data via a network. In one example embodiment, a client device120 may be a conventional computer system, such as a desktop or laptopcomputer. Alternatively, a client device 120 may be a device havingcomputer functionality, such as a personal digital assistant (PDA), amobile telephone, a smartphone, a tablet, or another suitable device.The vehicle 130 also may be capable of receiving client input as well astransmitting and/or receiving data via a network. In one embodiment, avehicle 130 contains a GPS device that can also communicate with themoving vehicle analysis system 101 via a network. The moving vehicleanalysis system 101 may convert the blobs to a visual representation ofmoving vehicles, deliver the visual representation of the movingvehicles to user device 120, output a visual representation of themoving vehicles to a user interface such as through graphical icons andvisual indicators, etc.

Example System Architecture

FIG. 2 illustrates a block diagram of an example system architecture fora moving vehicle analysis system 101. The system architecture shown inFIG. 2 may include an external system interface 201, an image store 102,an optional vehicular path detection module 105, an optional pixel blobidentification module 103, an optional feature extraction module 104, anoptional feature store 202, a machine learning model 106, a machinelearning training engine 203, a moving vehicle store 204, a trafficanalysis module 107, and a traffic pattern store 206. Optionalcomponents in FIG. 2 are shown using dashed lines.

The external system interface 201 shown in FIG. 2 may be a dedicatedhardware or software networking device that receives data packetsrepresenting images from the aerial imaging device 110. The externalsystem interface 201 may forward data packets representing reroutinginformation from the moving vehicle analysis system 101 via a network toclient devices and vehicles. In one example, the external systeminterface 201 forwards data packets at high speed along the opticalfiber lines of the Internet backbone. In another example, the externalsystem interface 201 exchanges routing information using the BorderGateway Protocol (BGP) and may be an edge router, a border router, or acore router.

The image store 102 shown in FIG. 2 may store low resolution imagesreceived from the aerial imaging device. The vehicular path detectionmodule 105, pixel blob identification module 103, and feature extractionmodule 104 may retrieve images stored by the image store 102 forprocessing. The image store 102 may be organized as a database or tableof images stored on one or more of removable or non-removable memorycards, tape cassettes, zip cassettes, and computer hard drives. In oneembodiment, the image store 102 may include multiple data fields, eachdescribing one or more attributes of the images. In one example, theimage store 102 contains, for a single image, the time of capture,spectral band information, geographical area coordinates, etc.

The vehicular path detection module 105 shown in FIG. 2 may perform edgeanalysis in the received images to identify pairs of edges, where eachpair of edges corresponds to a vehicular path. An example imagecontaining vehicular paths is illustrated and described below in detailwith reference to FIG. 8. A vehicular path may be a road, street,avenue, highway, unpaved road, etc. The vehicular path detection module105 may be configured to narrow down the search space for detectingmoving vehicles within the images from the image store 102 by selectinga vehicular path, e.g., roads, highways, or other paths, in which movingvehicles are expected.

The vehicular path detection module 105 shown in FIG. 2 may operate oneach pixel location (i, j) in an image. In one example embodiment, Srepresents an image and M represents the corresponding road map imageoutput. The function M(i, j) is defined to be l whenever location (i, j)in image S corresponds to a road pixel and 0 otherwise. The vehicularpath detection module 105 may identify points in an image at which thepixel attributes change sharply. The points at which pixel attributeschange sharply may be organized into a set of curved line segmentstermed edges. The vehicular path detection module 105 may perform threesteps in the edge analysis process to identify pairs of edges:filtering, enhancement, and detection. The filtering step reduces noisesuch as salt and pepper noise, impulse noise and Gaussian noise in theimages. The enhancement emphasizes pixels at locations (i, j) wherethere is a significant change in the pixel attribute value. In oneexample, the vehicular path detection module 105 performs enhancement bycomputing the gradient magnitude of the image at various pixel locations(i, j). The detection searches for pixel locations (i, j) that have agradient value higher than a threshold to detect edge pixels. Thevehicular path detection module 105 combines edges into edge pairs,corresponding to vehicular paths, based on the location of the pixelslying on each edge. In one example, a first edge of pixels is pairedwith a second edge of pixels into a vehicular path if the meandifference between the corresponding pixel locations of the two edges isless than the mean difference between the corresponding pixel locationsof the first edge and any other edge. Because each received image is ofthe same geographical region, the vehicular path detection module 105performs edge analysis on only one image and overlays the identifiedvehicular paths on the remaining images.

In alternative embodiments, the vehicular path detection module 105shown in FIG. 2 may analyze an image to create a probabilistic heat mapor blocked image containing an area of interest of vehicular paths wheremoving vehicles are expected. The vehicular path detection module 105may be further configured to incorporate other mapping sources, whichcontain geometric information (points, lines, and polygons). Thevehicular path detection module 105 may use the geometric information todirectly create the probabilistic heat map or in conjunction with otherimage processing operations such as a line finding algorithm, randomforest algorithm, Support Vector Machines (SVM), neural network,convolutional neural network (CNN), etc.

In one example embodiment, the pixel blob identification module 103shown in FIG. 2 may identify blobs of adjacent pixels using pixelclustering. Within an identified blob, adjacent pixels match each otherbased on a pixel attribute. In one example, for a grayscale image, thepixel attribute is a single number that represents the brightness of thepixel. In this example, the pixel attribute is a byte stored as an 8-bitinteger giving a range of possible values from 0 to 255. Zero representsblack and 255 represents white. Values in between 0 and 255 make up thedifferent shades of gray. In another example of color images, separatered, green and blue components are specified for each pixel. In thisexample, the pixel attribute is a vector of three numbers.

The pixel blob identification module 103 shown in FIG. 2 may identifypixel blobs in the images from the image store 102 received via theexternal system interface 201 by pixel clustering to identify adjacentpixels matching each other based on a pixel attribute, e.g., brightnessor color. In one example, the pixel blob identification module 103begins by initializing each pixel in an image as a region with theattribute of the pixel. The pixel blob identification module 103identifies two adjacent regions having the most similar attribute value.These two regions are merged to form a new region containing all thepixels of the two regions and having the attribute value as the averageof the attribute values of the two regions. The pixel blobidentification module 103 repeats the process until there are no similarregions left.

In another example embodiment, where each imaging sensor in aerialimaging device 110 corresponds to a different spectral band, blobs canappear at different locations in the red, blue, and green spectralbands, as illustrated and described below in detail with reference toFIG. 3. In this embodiment, the pixel blob identification module 103shown in FIG. 2 uses pixel clustering to identify pixel locations(matching each other based on a pixel attribute) where the attributevalue of the image corresponding to one band is significantly differentthan the other bands. In one example, a pixel at location (i, j) in thegreen band image is identified as part of a blob if G(i, j)>2×R(i, j),where G(i, j) is the attribute value of the pixel at location (i, j) inthe green band image and R(i, j) is the attribute value of the pixel atlocation (i, j) in the red band image. In this embodiment, the pixelblob identification module 103 can also be configured to process animage composed of different numbers of bands, where images correspondingto individual bands have been combined into a single multi-spectralimage. Individual pixel values are compared and analyzed for the samelocations across the different bands for the image.

In another example embodiment, the pixel blob identification module 103shown in FIG. 2 may use pixel clustering to examine a region or windowof pixels to detect blobs of adjacent pixels matching each other basedon a pixel attribute. In this embodiment, the pixel blob identificationmodule 103 receives an image from the image store 102, finds candidateregions to analyze, and analyzes the regions to determine which regionscontain blobs. In one example, a sliding window approach is used, wherea rectangular 3×3 pixel region continuously moves across the image toidentify candidate regions exhaustively. In this example, a blob isdetected when the average attribute value of pixels (matching each otherbased on the pixel attribute) within the region corresponding to oneband is significantly different than the other bands. In anotherembodiment, after candidate regions have been selected, the pixel blobidentification module 103 identifies “brighter” regions (matching eachother based on the pixel attribute) when compared to the surrounding“vehicular path” areas.

Other embodiments of the pixel blob identification module 103 shown inFIG. 2 may use one or a combination of the following: (a) regionproposal algorithms, such as Selective Search, which uses heuristics todetermine the best set of candidate window regions; (b) edge/cornerdetection methods, such as Harris Corner or Canny edge, which find edgesor corners in the image to use as candidate regions; (c) imagegradients, which extract edge strength information; (d) orientedfilters, which identify specific blob shapes; (e) thresholding methods,which use local or global threshold values to extract moving vehiclesblobs; (f) image patch descriptors such as Scale-Invariant FeatureTransform (SIFT), Speeded Up Robust Features (SURF), Features fromAccelerated Segment Test (FAST), Binary Robust Independent ElementaryFeatures (BRIEF), Fast Retina Keypoint (FREAK), and Histogram ofOriented Gradients (HOG), which calculate orientation and edgedescription features at a given image patch. An example low resolutionaerial image containing identified pixel blobs is illustrated anddescribed in FIG. 9.

Referring back to FIG. 2, the feature extraction module 104 may extracta feature vector from the received images describing the relativepositions of blobs with respect to the vehicular paths identified by thevehicular path detection module 105 and with respect to other blobs. Thefeature extraction module 104 reduces the redundancy in images, e.g.,repetitive pixel values, to transform an image into a reduced set offeatures (features vector). The feature vector contains the relevantinformation from the images, such that moving vehicles can be identifiedas pluralities of blobs by a machine learning model 106 by using thisreduced representation instead of the complete initial image. Thefeature vector may include at least one feature describing a relativeposition of a blob with respect to a pair of edges corresponding to avehicular path. Example features extracted by the feature extractionmodule are illustrated and described in FIG. 4 and FIG. 5. In oneembodiment, the following dimensionality reduction techniques are usedby the feature extraction module 104: independent component analysis,Isomap, Kernel PCA, latent semantic analysis, partial least squares,principal component analysis, multifactor dimensionality reduction,nonlinear dimensionality reduction, Multilinear Principal ComponentAnalysis, multilinear subspace learning, semidefinite embedding,Autoencoder, and deep feature synthesis.

The feature store 202 shown in FIG. 2 stores features extracted fromreceived images by the feature extraction module 104. The moving vehicleanalysis system 101 retrieves the stored features for training themachine learning model 106 as well as for associating moving vehicleswith pluralities of blobs. Details of features stored in the featurestore 202 are provided in FIG. 4 and FIG. 5.

The moving vehicle analysis system 101 may train the machine learningmodel 106 using training sets and data from the feature store 202. Inone embodiment, the machine learning model 106 receives training setsincluding labeled blobs corresponding to moving vehicles or labelededges corresponding to vehicular paths. The machine learning trainingengine 203 shown in FIG. 2 trains the machine learning model 106 usingtraining sets to determine a score for two or more blobs. The score isindicative of a likelihood that the blobs corresponds to a movingvehicle based on the feature vector. Training sets are illustrated inFIG. 5. The process followed by the machine learning training engine 203is illustrated in FIG. 4. The moving vehicle analysis system 101 selectsblobs based on whether the score exceeds a threshold and associates theblobs with a moving vehicle.

In alternative embodiments, the machine learning model 106 shown in FIG.2 (in the form of a convolutional neural network) may generate anoutput, without the need for feature extraction, edge analysis or blobidentification, directly from the images. A CNN is a type offeed-forward artificial neural network in which the connectivity patternbetween its neurons is inspired by the organization of a visual cortex.Individual cortical neurons respond to stimuli in a restricted region ofspace known as the receptive field. The receptive fields of differentneurons partially overlap such that they tile the visual field. Theresponse of an individual neuron to stimuli within its receptive fieldcan be approximated mathematically by a convolution operation. CNNs arebased on biological processes and are variations of multilayerperceptrons designed to use minimal amounts of preprocessing. Advantagesof CNNs include the obviation of feature extraction and the use ofshared weight in convolutional layers, which means that the same filter(weights bank) is used for each pixel in the layer; this both reducesmemory footprint and improves performance. The output of the machinelearning model 106 (in the form of a CNN) may include blobs, where eachblob includes one or more adjacent pixels in a distinct image of theimages, where the adjacent pixels match each other based on a pixelattribute. The output may include a score indicative of a likelihoodthat the blobs correspond to a moving vehicle. The output may includeone or more pixels locations corresponding to a blob. The output mayinclude pixel locations corresponding to a vehicular path. The outputmay include the number of pixels in each blob. The output may include anassociation between the blobs and a moving vehicle.

The moving vehicle store 204 shown in FIG. 2 may store informationrepresenting moving vehicles received from the machine learning model106. The moving vehicle store 204 may be organized as a database, table,or file stored on one or more of removable or non-removable memorycards, tape cassettes, zip cassettes, and computer hard drives. In oneembodiment, the moving vehicle store 204 stores multiple data fields,each describing one or more attributes of the moving vehiclesidentified. In one example, the moving vehicle store 204 stores thecount of the number of moving vehicles identified in the receivedimages, and for a single vehicle, the time of capture, geographicalregion coordinates, diameter of each blob in pixels, size of thevehicle, speed and direction of the vehicle, etc.

The traffic analysis module 107 shown in FIG. 2 receives informationfrom the moving vehicle store 204 and analyzes it on a per-vehiclebasis, a per-vehicular path basis, or on a geographical region basis. Inone example embodiment, the traffic analysis module 107 receivesinformation corresponding to the pixel resolution of the images from theimage store 202. The pixel resolution of low resolution images isgenerally measured in GSD units of 3-5 meters/pixel. In this embodiment,the traffic analysis module 107 also receives information correspondingto the number of pixels in each blob corresponding to the vehicle fromthe moving vehicle store 204. In one example, r represents the pixelresolution in meters/pixel, d(k) represents the diameter of a blob kcorresponding to a vehicle v, and n represents the number of blobscorresponding to vehicle v. The traffic analysis module 107 determinesthe size of vehicle v as (Σ_(k) d(k))/n×r, where (Σ_(k) d(k))/n is theaverage diameter in pixels of all blobs corresponding to vehicle v.

In another example embodiment, the traffic analysis module 107 shown inFIG. 2 determines the lengths of vehicular paths in the images. In thisexample embodiment, the traffic analysis module 107 receives informationcorresponding to the pixel resolution of the images from the image store202. The traffic analysis module 107 may determine the number h ofpixels associated with a length l of a vehicular path by assigning alabel to every pixel in the path and counting the labeled pixels alongthe edges of the path. The traffic analysis module 107 may determine thelength l of the vehicular path based on the pixel resolution and thenumber of pixels h associated with the length of the vehicular path. Inone example, r represents the pixel resolution in meters/pixel, h1represents the number of pixels associated with the first edge of avehicular path, and h2 represents the number of pixels associated withthe second edge of the vehicular path. The traffic analysis module 107determines the length l of the vehicular path as l=(h1+h2)/2×r, where(h1+h2)/2 is the average number of pixels associated with the length ofthe vehicular path.

In another example embodiment, the traffic analysis module 107 shown inFIG. 2 determines the density of moving vehicles on a vehicular pathbased on the count of the moving vehicles and the length of thevehicular path. In one example, c represents the count of the movingvehicles and l represents the length of the vehicular path. The trafficanalysis module 107 determines the density of moving vehicles as c/l.

In another example embodiment, the traffic analysis module 107 shown inFIG. 2 determines a directional vector for the corresponding blobsassociated with a moving vehicle by determining a difference between acentroid of each blob of the corresponding blobs, as illustrated anddescribed below in detail with reference to FIG. 11. The trafficanalysis module 107 determines the centroid or geometric center of ablob by the arithmetic mean location of all the pixel locations in theblob as Σ_(ij)(i, j)/m, where (i, j) represents the location of a pixelin the blob and m represents the number of pixels in the blob. In oneexample, there are two blobs a and b corresponding to a moving vehicle,each blob corresponding to one image. In this example, (i, j) representsthe location of a pixel in blob a, and m represents the number of pixelsin blob a. Further, (p, q) represents the location of a pixel in blob band s represents the number of pixels in blob b. The traffic analysismodule 107 determines the directional vector as Σ_(ij)(i, j)/m−(p, q)/s.This process is illustrated in FIG. 11. The directional vectorcorresponds to a speed and a direction of the moving vehicle.

The traffic analysis module 107 shown in FIG. 2 may determine the lengthof the directional vector as r×(Σ_(ij)(i, j)/m−Σ_(pq) (p, q)/s), where ris the pixel resolution. In another example, t_(a) represents the timeof capture of the image containing blob a and t_(b) represents the timeof capture of the image containing blob b. In this example, the trafficanalysis module 107 determines the speed of the moving vehicle by thelength of the directional vector divided by the difference in timebetween images a and b as (r×Σ_(ij)(i, j)/m−Σ_(pq)(p,q)/s))/(t_(a)−t_(b)), as illustrated and described below in detail withreference to FIG. 12. In another example, the traffic analysis module107 determines the direction of the moving vehicle by tan⁻¹ (q−j/p−i),where (q−j/p−i) is the ratio of the difference between the Yco-ordinates of the two centroids to the difference between the Xco-ordinates of the two centroids.

In another example embodiment, the traffic analysis module 107 shown inFIG. 2 may determine the average speed of the moving vehicles identifiedin the received images as the sum of the speeds of the one or moremoving vehicles divided by the count of the one or more moving vehicles.The traffic analysis module 107 analyzes a traffic pattern comprising atime of capture of an image, the count of the moving vehicles on avehicular path, the size of the moving vehicles, the average speed, thedensity of moving vehicles, and the length of the vehicular path. If theaverage speed falls below a threshold or the count or density of movingvehicles exceeds a threshold, the moving vehicle analysis system 101 maysend rerouting information to a client device 120 or a vehicle 130 viathe external system interface 201. In one example, the reroutinginformation may be determined using Dijkstra's algorithm, theBellman-Ford algorithm, A* search, Floyd-Warshall algorithm, Johnson'salgorithm, Viterbi algorithm, etc.

The traffic pattern store 206 shown in FIG. 2 may store traffic patternsreceived from the traffic analysis module 107. The traffic pattern store206 may be organized as a database or table stored on one or more ofremovable or non-removable memory cards, tape cassettes, zip cassettes,and computer hard drives. In one embodiment, the traffic pattern store206 stores multiple data fields, each describing one or more attributesof a vehicular path. In one example, the traffic pattern store 206stores, for a single vehicular path, the time of capture of images,geographical region coordinates, the count of the one or more movingvehicles on the vehicular path, the size of each of the one or moremoving vehicles, the average speed, the density of moving vehicles, thelength of the vehicular path, etc.

Low Resolution Aerial Image

Referring now to FIG. 3, it illustrates an example low resolution aerialimage 300 combined from three red, blue, and green imaging sensorswithin aerial imaging device 110, each sensor capturing a distinct imagein a different band. Combined image 300 includes a blue blob 301, greenblob 301, red blob 303, and a vehicular path 304. The three imagingsensors within aerial imaging device 110 capture distinct images atsmall time differences. The captured images appear the same whencombined into image 300 since most objects within the image are notmoving (e.g., vehicular path 304, trees, houses, etc.). However, movingvehicles do not line up perfectly because of the difference in time ofcapture when the different images are combined. For example, movingvehicles can appear as blobs at different locations in the red, blue,and green band. This is apparent when the three images are combined intothe single image 300. In image 300, blob 301, shown in FIG. 3 as acollection of blue-colored pixels, is captured by the blue spectrumimaging sensor. In image 300, blob 302, shown in FIG. 3 as a collectionof green-colored pixels, is captured by the green spectrum imagingsensor. In image 300, blob 303, shown in FIG. 3 as a collection ofred-colored pixels, is captured by the red spectrum imaging sensor. Theblue blob 301 and green blob 302 are closer together in image 300because the time delay between those two spectral bands is smallercompared to the time delay between the blue and red spectral bands orthe time delay between the green and red spectral bands.

The vehicular path pixels may be identified as described in detail withreference to FIG. 2. In alternative embodiments, the aerial imagingdevice 110 may capture images from the infrared and panchromatic bands.The aerial imaging device 110 can be further configured to captureimages from a single band at distinct times. The obtained images aretransmitted to the moving vehicle analysis system 101 via the externalsystem interface 201.

Example Machine Learning Training Process

FIG. 4 illustrates an example training process for the machine learningtraining engine 203 for the machine learning model 106 in the movingvehicle analysis system 101. The process may use training images 401,the pixel blob identification module 103, the image store 102, thevehicular path detection module 105, the feature extraction module 104,a feature vector 410, the machine learning model 106, the trafficdatabase 109, and training labels 402. Optional elements are shown usingdashed lines in FIG. 4. In one embodiment, the pixel blob identificationmodule 103 shown in FIG. 4 may perform pixel clustering to identifyblobs by, for each image of the training images 401, identifying blobscomprising adjacent pixels, wherein the adjacent pixels match each otherbased on a pixel attribute. The pixel blob identification module 103 maysend the training images 401 containing identified blobs to the featureextraction module 104.

The image store 102 shown in FIG. 4 also receives training images 401and transmits them to the vehicular path detection module 105. Thevehicular path detection module 105 performs edge analysis, asillustrated and described in detail above in FIG. 2, in the trainingimages 401 to identify pairs of edges, each pair of edges correspondingto a vehicular path. The feature extraction module 104 shown in FIG. 4extracts a feature vector 410 from the training images 401 and thevehicular paths. The features corresponding to the training images 401are used for training the machine learning model 106 based on traininglabels 402. Details of training labels 402 are illustrated and describedin FIG. 5.

The feature vector 410 shown in FIG. 4 may have at least one featurerepresenting a relative position of a blob within an image. In oneexample embodiment, in which the pixel blob identification module 103identifies blobs, a feature 410 a may represent whether a blob islocated between the pair of edges corresponding to the vehicular path;this feature teaches the machine learning model 106 that the blob mayrepresent a moving vehicle on a vehicular path. FIG. 4 and the otherfigures use like reference numerals to identify like elements. A letterafter a reference numeral, such as “410 a,” indicates that the textrefers specifically to the element having that particular referencenumeral. A reference numeral in the text without a following letter,such as “410,” refers to any or all of the elements in the figuresbearing that reference numeral, e.g., “410” in the text refer toreference numerals “410 a” and/or “410 b” in the figures.

A feature 410 b shown in FIG. 4 may represent whether two blobsintersect each other; this feature teaches the machine learning model106 that the two blobs may not represent a moving vehicle becausevehicles cannot intersect. A feature 410 c may represent whether adifference between the centroid of each blob of a group of blobs matchesa labeled direction of the vehicular path in the training labels 402;this feature teaches the machine learning model 106 that the blobs mayrepresent a moving vehicle because it is moving in the correctdirection. A feature 410 d may represent whether blobs are located atthe same pixel locations in more than one image; this feature teachesthe machine learning model 106 that the blobs may not represent movingvehicles because the blobs did not move locations between imagescaptured at different times.

In alternative embodiments, the machine learning model 106 shown in FIG.4 (in the form of a CNN) may generate an output directly from theimages, without the need for feature extraction, edge analysis or blobidentification. The machine learning model 106 shown in FIG. 4 (in theform of a CNN) may identify pixel blobs within images, as illustratedand described in detail above in FIG. 2. In such embodiments, a feature410 e may represent an association between pixel locations and a pixelattribute. For example, a feature may represent the brightness value ofa pixel relative to pixels located on its right in an image; thisfeature teaches the machine learning model 106 that the pixel may bepart of a blob representing a moving vehicle because the pixel isbrighter than surrounding pixels. In another example, a feature mayrepresent the brightness of a pixel relative to the average brightnessof pixels located on the same row in an image; this feature teaches themachine learning model 106 that the pixel may be part of a blobrepresenting a moving vehicle because the pixel is brighter thansurrounding pixels.

The machine learning training engine 203 may trains the machine learningmodel 106 shown in FIG. 4 using the feature vector 410 and traininglabels 402. In one embodiment, the machine learning model 106 is therebyconfigured to determine a score for each pixel location in an image, thescore indicative of a likelihood that the pixel location corresponds toa blob. In another embodiment, the machine learning model 106 isconfigured to determine a score for blobs, the score indicative of alikelihood that the blobs correspond to a moving vehicle. In alternativeembodiments, the machine learning model 106 is configured to generate anoutput including blobs and a score indicative of a likelihood that theblobs corresponds to a moving vehicle. In an embodiment, the machinelearning model 106 is configured to generate an output including one ormore pixels locations corresponding to a blob and a score indicative ofa likelihood that the pixels locations correspond to a blob. In anembodiment, the machine learning model 106 is configured to generate anoutput including pixel locations corresponding to a vehicular path and ascore indicative of a likelihood that the pixel locations correspond toa vehicular path. In an embodiment, the machine learning model 106 isconfigured to generate an output including a number of pixels in eachidentified blob. In an embodiment, the machine learning model 106 isconfigured to generate an output including an association between theidentified blobs and a moving vehicle.

The machine learning model training engine 203 shown in FIG. 4 may applymachine learning techniques to train the machine learning model 106 thatwhen applied to features outputs indications of whether the featureshave an associated property or properties, e.g., that when applied tofeatures of received images outputs estimates of whether there aremoving vehicles present, such as probabilities that the features have aparticular Boolean property, or an estimated value of a scalar property.As part of the training of the machine learning model 106, the machinelearning training engine 203 forms a training set of features 410 andtraining labels 402 by identifying a positive training set of featuresthat have been determined to have the property in question (presence ofmoving vehicles), and, in some embodiments, forms a negative trainingset of features that lack the property in question, as described belowin detail with reference to FIG. 5. For example, each training set mayinclude labeled blobs corresponding to a moving vehicle. The machinelearning training engine 203 may applies dimensionality reduction (e.g.,via linear discriminant analysis (LDA), principle component analysis(PCA), or the like) to reduce the amount of data in the feature vector410 to a smaller, more representative set of data.

The machine learning training engine 203 may use supervised machinelearning to train the machine learning model 106 shown in FIG. 4, withthe feature vectors 410 of the positive training set and the negativetraining set serving as the inputs. In other embodiments, differentmachine learning techniques, such as linear support vector machine(linear SVM), boosting for other algorithms (e.g., AdaBoost), logisticregression, naïve Bayes, memory-based learning, random forests, baggedtrees, decision trees, boosted trees, boosted stumps, neural networks,CNNs, etc., may be used. The machine learning model 106, when applied tothe feature vector 410 extracted from a set of received images, outputsan indication of whether blobs has the property in question, such as aBoolean yes/no estimate, or a scalar value representing a probability.

In some example embodiments, a validation set is formed of additionalfeatures, other than those in the training sets, which have already beendetermined to have or to lack the property in question. The machinelearning training engine 203 applies the trained machine learning model106 shown in FIG. 4 to the features of the validation set to quantifythe accuracy of the machine learning model 106. Common metrics appliedin accuracy measurement include: Precision=TP/(TP+FP) andRecall=TP/(TP+FN), where precision is how many the machine learningmodel 106 correctly predicted (TP or true positives) out of the total itpredicted (TP+FP or false positives), and recall is how many the machinelearning model 106 correctly predicted (TP) out of the total number offeatures that did have the property in question (TP+FN or falsenegatives). The F score (F-score=2×PR/(P+R)) unifies precision andrecall into a single measure. In one embodiment, the machine learningtraining engine 203 iteratively re-trains the machine learning model 106until the occurrence of a stopping condition, such as the accuracymeasurement indication that the machine learning model 106 issufficiently accurate, or a number of training rounds having takenplace.

In alternative embodiments, the machine learning model 106 may be a CNNthat learns useful representations (features) such as which blobscorrespond to moving vehicles directly from training sets withoutexplicit feature extraction. For example, the machine learning model 106may be an end-to-end recognition system (a non-linear map) that takesraw pixels from the training images 401 directly to internal labels.

In an embodiment, the machine learning model 106 may use informationabout traffic patterns from the traffic database 109 in which databecomes available in a sequential order and is used to update themachine learning model 106's best predictor for future data at eachstep. In an example, the traffic database 109 may be used to dynamicallyadapt the machine learning model 106 to new patterns in the trafficdatabase 109 as a function of time. In this embodiment, the machinelearning training engine 203 may use statistical learning models (e.g.stochastic gradient descent, perceptrons) or adversarial models.

Example Training Sets for the Machine Learning Model

FIG. 5 illustrates example training sets 500 and 510 for the machinelearning model 106 in the moving vehicle analysis system 101. Eachtraining set includes the training images 401 and training labels 402illustrated and described in FIG. 4. The positive training set 500 shownin FIG. 5 contains features 410 that have been determined to have thepresence of moving vehicles. The positive training set 500 is a combinedimage from images taken at distinct times. The positive training set 500includes two labeled pixel blobs 501 and 502. In one example, pixel blob501 corresponds to the green spectral band and pixel blob 502corresponds to the red spectral band. The positive training set 500further includes a moving vehicle training label 503 used by the machinelearning model training engine 203 to train the machine learning model106 that the two pixel blobs 501 and 502 correspond to a moving vehicle.The positive training set 500 further includes a vehicular path edgetraining label 504 used by the machine learning training engine 203 totrain the machine learning model 106 that the edge corresponds to avehicular path.

The negative training set 510 shown in FIG. 5 contains features 410 thathave been determined to lack the presence of moving vehicles. Thenegative training set 510 is also a combined image from images taken ata distinct time. The negative training set 510 includes three unlabeledpixel blobs 511, 512, and 513, as well as two unlabeled edges 514 and515. In one example, pixel blob 511 corresponds to the green spectralband and pixel blob 512 corresponds to the red spectral band. Pixel blob513 may correspond to the blue spectral band. Pixel blob 511 is locatedin a background region 516 that is not between the two edges 514 and 515and does not correspond to a moving vehicle. Pixel blob 511 is notlabeled as a moving vehicle.

Pixel blob 512 is located between the unlabeled pair of edges 514 and515. In addition, while pixel blob 512 corresponds to the red spectralband, there is no corresponding pixel blob from another spectral bandpresent between the unlabeled pair of edges 514 and 515. Therefore,pixel blob 512 does not correspond to a moving vehicle. Pixel blob 512is not labeled as a moving vehicle. Pixel blob 513 is located in abackground region 517 that is not between the two edges 514 and 515 anddoes not correspond to a moving vehicle. Pixel blob 513 is not labeledas a moving vehicle. The negative training set 510 is used by themachine learning training engine 203 to train the machine learning model106 that the three pixel blobs 511, 512, and 513 do not correspond tomoving vehicles. The negative training set 510 is further used by themachine learning training engine 203 to train the machine learning model106 that the edges 514 and 515 do not correspond to a vehicular path.

In an embodiment, the training sets are created by manually labelingpluralities of blobs that represent high scores and pluralities of blobsthat represent low scores. In another embodiment, the machine learningtraining engine 203 extracts training sets from stored images obtainedfrom the image store 102. In one example, if a stored image containsblobs located between a pair of edges corresponding to a vehicular path,the machine learning training engine 203 uses the blobs as a positivetraining set. If a stored image contains a blob not located between apair of edges corresponding to a vehicular path, the machine learningtraining engine 203 uses the blob as a negative training set.

Example Process

FIG. 6 illustrates an example process for the moving vehicle analysissystem 101. In some embodiments, the process may have different and/oradditional steps than those described in conjunction with FIG. 6. Forexample, optional steps are shown using dashed lines in FIG. 6. Steps ofthe process may be performed in different orders than the orderdescribed in conjunction with FIG. 6. Some steps may be executed inparallel. Alternatively, some of the steps may be executed in paralleland some steps executed sequentially. Alternatively, some steps mayexecute in a pipelined fashion such that execution of a step is startedbefore the execution of a previous step.

The moving vehicle analysis system 101 receives 600 images of ageographical area taken at distinct times from an aerial imaging device110 and stores the images in an image store 102 as illustrated anddescribed above in detail with reference to FIG. 2. An example image ofa geographical area received from an aerial imaging device 110 isillustrated and described below in detail with reference to FIG. 8.

In one example embodiment, the pixel blob identification module 103 mayperform pixel clustering according to one of the methods described inFIG. 2 to identify 604 blobs in the received images, where each blobincludes adjacent pixels matching each other based on a pixel attribute.The feature extraction module 104 may extract 608 a feature vector 410from the images as illustrated and described above in detail withreference to FIG. 4. The feature vector 410 includes at least a featuredescribing a relative position of a blob.

The moving vehicle analysis system 101 may transmit the feature vector410 as input to the machine learning model 106. The feature vector 410may be physically transmitted as a digital bit stream or a digitizedanalog signal over a point-to-point or point-to-multipoint communicationchannel, e.g., copper wires, optical fibers, wireless communicationchannels, storage media and computer buses. The feature vector 410 maybe represented as an electromagnetic signal, such as an electricalvoltage, radiowave, microwave, or infrared signal.

In one embodiment, the machine learning model 106 generate 612 anoutput, based on the feature vector 410, including blobs (each blobcorresponding to a distinct image) and a score indicative of alikelihood that the blobs correspond to a moving vehicle. In anotherembodiment, in which the machine learning model 106 is a CNN, themachine learning model 106 generate 612 an output, based on the images,including blobs and a score indicative of a likelihood that the blobscorrespond to a moving vehicle. Based on the generated score exceeding athreshold, the moving vehicle analysis system 101 stores 616 anassociation between blobs and a moving vehicle. The association betweenblobs and a moving vehicle may be stored as a database, a lookup table,or a list of data pointers, etc., that is stored in static programstorage, pre-fetched by the moving vehicle analysis system 101, orstored in hardware on an application-specific platform. The movingvehicle analysis system 101 determines 620 a count of the number ofmoving vehicles in the images, where each moving vehicle is associatedwith corresponding blobs.

Example Application of the Process

FIG. 7 illustrates an example application of the process illustrated anddescribed above in detail with reference to FIG. 6 for the movingvehicle analysis system 101 for processing low resolution aerialimagery. The process may involve images 701, an optional pixel blobidentification module 103, an image store 102, a vehicular pathdetection module 105, a feature vector 410, a machine learning model106, a moving vehicle store 204, a traffic analysis module 107, atraffic database 109 and a traffic pattern store 205. Optional elementsare shown using dashed lines in FIG. 7. The moving vehicle analysissystem 101 receives images 701 of a geographical area via an externalsystem interface 201, each image 701 captured at a distinct time. Theimages 701 may correspond to one or more spectral bands. The receivedimages 701 shown in FIG. 7 are stored in the image store 102, asdescribed above in detail with reference to FIG. 2.

In one embodiment, the pixel blob identification module 103 shown inFIG. 7 performs pixel clustering as described in FIG. 2 to identifypixel blobs, each blob comprising adjacent pixels, wherein the adjacentpixels match each other based on a pixel attribute, such as color orbrightness. The vehicular path detection module 105 performs edgeanalysis, as described, in the images to identify pairs of edges, thepairs of edges corresponding to vehicular paths. The moving vehicleanalysis system 101 extracts a feature vector 410 from the images to beused by a machine learning model 106. In one embodiment described inFIG. 4, the features describe the relative positions of blobs withrespect to vehicular paths and with respect to other blobs. In anotherembodiment described in FIG. 4, the feature vector 410 describes thebrightness of a pixel relative to the average brightness of other pixelslocated on the same row in an image.

The moving vehicle analysis system 101 transmits the extracted featurevector 410 to a machine learning model. In one embodiment, the machinelearning model is a machine learning model 106 shown in FIG. 7. In oneexample embodiment, the machine learning model 106 may generate a scoreindicative of a likelihood that pixel blobs corresponds to a movingvehicle. In another embodiment in which the machine learning model 106is a CNN, the score is indicative of a likelihood that a pixel in animage 701 corresponds to a blob representing a moving vehicle. In thisembodiment, the machine learning model 106 is configured to optimize theconditional probability that a pixel in an image 701 corresponds to ablob representing a moving vehicle based on the pixel's location andbrightness in the image 701. In one example, P(blob(e)) represents theprobability that a pixel having a brightness attribute e corresponds toa blob. In this example, P(vehicle(i, j)) represents the probabilitythat a pixel location (i, j) within image 701 corresponds to a movingvehicle. The machine learning model 106 is configured to optimize thesum Σ_(i) Σ_(i) Σ_(e) P(vehicle(i, j)|blob(e)), which represents the sumof conditional probabilities over all pixel locations (i, j) in image701 and all values of pixel brightness e that a pixel location (i, j)within image 701 corresponds to a moving vehicle, given that the pixelhas a brightness attribute e.

The traffic analysis module 107 shown in FIG. 7 may determine a count ofone or more moving vehicles in the images, each moving vehicleassociated with corresponding blobs, and stores information about theidentified moving vehicles in the moving vehicle store 204 as describedabove in detail with reference to FIG. 2. The traffic analysis module107 may use information from the traffic database 109 such asintersection counts (directional volume of traffic as it moves through asignalized or unsignalized intersection, the number of left-turn,right-turn, U-turn and through movements, etc.), roundabout counts (theturning movement volume of roundabouts, the entry, circulating and exitflow rate for each roundabout leg, etc.), road volume data (vehiclevolume, vehicle classification, lane-by-lane classifications andvolumes, etc.), vehicle gap data (the number of gaps in seconds thatexist between vehicles in the major traffic stream at specific referencepoints), etc., to compare with positions of identified moving vehiclesand create accurate routing, estimated time of arrival, etc.

Example Image of a Geographical Area

FIG. 8 is an example low resolution aerial image 800 of a geographicalarea combined from multiple imaging sensors within the aerial imagingdevice 110, as described above in detail with reference to FIG. 1. Image800 is received by the moving vehicle analysis system 101, as describedabove in detail with reference to FIG. 2. The multiple imaging sensorswithin the aerial imaging device 110 correspond to different spectralbands and capture images at distinct times, as described above in detailwith reference to FIG. 1. Vehicular paths 801, 802, and 803 detected bythe vehicular path detection module 105 or the machine learning model106 (in the form of a CNN) are visible in the image 800. Blobs 804 onvehicular path 801 correspond to a moving vehicle as identified by themachine learning model 106. Blobs 805 on vehicular path 801 correspondto another moving vehicle as identified by the machine learning model106. The direction of travel 806 corresponds to the direction from thegreen to the red blobs within corresponding blobs 804 and 805 becausethe green spectral band image is captured before the red spectral bandimage.

Example Image Containing Pixel Blobs

FIG. 9 illustrates an example low resolution aerial image 900 combinedfrom multiple imaging sensors and containing pixel blobs and vehicularpaths identified by the moving vehicle analysis system 101, as describedabove in detail with reference to FIG. 2. In image 900, the movingvehicle analysis system 101 has identified background regions 901 and902 containing no edges corresponding to vehicular paths. The pixelblobs 910 and 909 shown in FIG. 9 have been identified as describedabove in detail with reference to FIG. 2. The pixel blobs 910 and 909are located in background regions 901 and 902 containing no edges andtherefore are not identified as corresponding to moving vehicles, asdescribed above in detail with reference to FIG. 7.

In image 900, the moving vehicle analysis system 101 has also identifiededges 904 and 905 corresponding to a vehicular path 903, as describedabove in detail with reference to FIG. 2. Only blobs located between theedges 904 and 905 may correspond to moving vehicles. The pixel blobs906, 907, and 908 shown in FIG. 9 have been identified. The pixel blobs906, 907, and 908 correspond to images taken at distinct times and arelocated between the edges 904 and 905 corresponding to the vehicularpath 903. Therefore, the pixel blobs 906, 907, and 908 may be identifiedas corresponding to moving vehicles by the application of the processdescribed above in detail with reference to FIG. 7.

Example Output of Traffic Analysis

FIG. 10 illustrates an example output 1000 of the traffic analysismodule 107. The image illustrated in FIG. 10 contains a vehicular path1001 and two pixel blobs 1002 and 1003 that correspond to a movingvehicle identified by the application of the process described abovewith reference to FIG. 7. The traffic analysis module 107 determines adirectional vector for the blobs corresponding to each moving vehicle bydetermining a difference between a centroid of each of the correspondingblobs. As described above in detail with reference to FIG. 2, in theexample output 1000 of the traffic analysis module 107 shown in FIG. 10,the traffic analysis module 107 has determined the centroid 1004 of theblob 1002 as the arithmetic mean location of all the pixel locations inthe blob 1002 as Σ_(ij) (i, j)/m, where (i, j) represents the pixellocation of a pixel in the blob 1002 and m represents the number ofpixels in the blob 1002.

Further, in the example output 1000, the traffic analysis module 107 hasdetermined the centroid 1005 of the blob 1003 as the arithmetic meanlocation of all the pixel locations in the blob 1003 as Σ_(pq) (p, q)/s,where (p, q) represents the pixel location of a pixel in the blob 1003and s represents the number of pixels in the blob 1003. In exampleoutput 1000, the traffic analysis module 107 has determined thedirectional vector 1006 of the moving vehicle as Σ_(ij) (i,j)/m−Σ_(pq)(i, j)/s. The directional vector corresponds to a speed and adirection of the moving vehicle.

Example Input Images from a Single Imaging Sensor

FIG. 11 illustrates example low resolution aerial images 1100 and 1110,each captured from a single imaging sensor at distinct times t₁ and t₂,where t₁ is earlier than t₂ in time. The images 1100 and 1110 arereceived as input by the moving vehicle analysis system 101. In oneexample the image 1100 captured at time t₁ corresponds to the greenspectral band. In this example, the image 1110 captured at time t₂corresponds to the red spectral band. The image 1100 includes avehicular path 1101 and two edges 1102 and 1103 corresponding to thevehicular path 1101. The image 1100 further includes two pixel blobs1104 and 1105 corresponding to the green spectral band. Both blobs 1104and 1105 are located between the pair of edges 1102 and 1103corresponding to the vehicular path 1101. Pixel blob 1104 is locatedcloser to edge 1102 than to edge 1103, and blob 1105 is located closerto edge 1103 than to edge 1102.

The image 1110 shown in FIG. 11 also includes the vehicular path 1101and the two edges 1102 and 1103 corresponding to the vehicular path1101. The image 1110 further includes two pixel blobs 1114 and 1115corresponding to the red spectral band. Both blobs 1114 and 1115 arelocated between the pair of edges 1102 and 1103 corresponding to thevehicular path 1101. Pixel blob 1114 is located closer to edge 1102 thanto edge 1103, and blob 1115 is located closer to edge 1103 than to edge1102. Because image 1100 is captured at time t₁ which is earlier thanthe time t₂ at which image 1110 is captured, the blobs 1104 and 1105represent positions of moving vehicles earlier in time. Because image1110 is captured at time t₂ which is later than the time t₁ at whichimage 1100 is captured, the blobs 1114 and 1115 represent positions ofmoving vehicles later in time.

Example Output of a Machine Learning Model

FIG. 12A illustrates an example output 1200 of the machine learningmodel 106 for the received images 1100 and 1110 shown in FIG. 11according to the process of FIG. 6. The output 1200 of the machinelearning model 106 includes the identified vehicular path 1101 shown inFIG. 12A. The output 1200 of the machine learning model 106 furtherincludes the blobs 1104 and 1114 from images 1100 and 1110 locatedwithin the edges corresponding to vehicular path 1101. Because blob 1104corresponds to the green spectral band and blob 1114 corresponds to thered spectral band, the machine learning model 106 has identified thatblobs 1104 and 1114 from images 1100 and 1110 correspond to a movingvehicle 1204 driving closer to the upper edge of vehicular path 1101.

The output 1200 of the machine learning model 106 further includes theblobs 1105 and 1115 from images 1100 and 1110 located between the pairof edges corresponding to vehicular path 1101 shown in FIG. 12A. Becauseblob 1105 corresponds to the green spectral band and blob 1115corresponds to the red spectral band, the machine learning model 106 hasidentified that blobs 1105 and 1115 from images 1100 and 1110 correspondto a second moving vehicle 1205 driving closer to the lower edge ofvehicular path 1101.

Example Output of Traffic Analysis

FIG. 12B illustrates an example output 1210 of a traffic analysis module107 for the two moving vehicles 1204 and 1205 from FIG. 12A. In FIG.12B, the traffic analysis module 107 has computed a directional vector1214 for moving vehicle 1204 from FIG. 12A, as described above in detailwith reference to FIG. 2. The moving vehicle 1204 is represented in FIG.12A by blob 1104 corresponding to the green spectral band captured attime t₁ and by blob 1114 corresponding to the red spectral band capturedat time t₂, where t₁ is earlier than t₂. Therefore, the traffic analysismodule 107 computes the directional vector 1214 shown in FIG. 12B of themoving vehicle 1204 from FIG. 12A as moving in the direction from blob1104 to blob 1114.

In FIG. 12B, the traffic analysis module 107 has further computed adirectional vector 1215 for moving vehicle 1205 from FIG. 12A, asdescribed above in detail with reference to FIG. 2. The moving vehicle1205 is represented in FIG. 12A by blob 1105 corresponding to the greenspectral band captured at time t₁ and by blob 1115 corresponding to thered spectral band captured at time t₂, where t₁ is earlier than t₂.Therefore, the traffic analysis module 107 computes the directionalvector 1215 shown in FIG. 12B of the moving vehicle 1205 from FIG. 12Aas moving in the direction from blob 1105 to blob 1115.

Example Machine Architecture

FIG. 13 is a block diagram illustrating components of an example machineable to read instructions described as processes herein from amachine-readable medium and execute them in at least one processor (orcontroller). Specifically, FIG. 13 shows a diagrammatic representationof a machine in the example form of a computer system 1300. The computersystem 1300 can be used to execute instructions 1324 (e.g., program codeor software) for causing the machine to perform any one or more of themethodologies (or processes) described herein. In alternativeembodiments, the machine operates as a standalone device or a connected(e.g., networked) device that connects to other machines. In a networkeddeployment, the machine may operate in the capacity of a server machineor a client machine in a server-client network environment, or as a peermachine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, a network router, switch or bridge,or any machine capable of executing instructions 1324 (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute instructions 1324 to perform any one or more of themethodologies discussed herein.

The example computer system 1300 includes one or more processing units(generally processor 1302). The processor 1302 is, for example, acentral processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), a controller, a state machine, one ormore application specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese. The computer system 1300 also includes a main memory 1304. Thecomputer system may include a storage unit 1316. The processor 1302,memory 1304 and the storage unit 1316 communicate via a bus 1308.

In addition, the computer system 1300 can include a static memory 1306,a display driver 1310 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), or a projector). The computer system 1300may also include alphanumeric input device 1312 (e.g., a keyboard), acursor control device 1314 (e.g., a mouse, a trackball, a joystick, amotion sensor, or other pointing instrument), a signal generation device1318 (e.g., a speaker), and a network interface device 1320, which alsoare configured to communicate via the bus 1308.

The storage unit 1316 includes a machine-readable medium 1322 on whichis stored instructions 1324 (e.g., software) embodying any one or moreof the methodologies or functions described herein. The instructions1324 may also reside, completely or at least partially, within the mainmemory 1304 or within the processor 1302 (e.g., within a processor'scache memory) during execution thereof by the computer system 1300, themain memory 1304 and the processor 1302 also constitutingmachine-readable media. The instructions 1324 may be transmitted orreceived over a network 1326 via the network interface device 1320.

While machine-readable medium 1322 is shown in an example embodiment tobe a single medium, the term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 1324. The term “machine-readable medium” shall also betaken to include any medium that is capable of storing instructions 1324for execution by the machine and that cause the machine to perform anyone or more of the methodologies disclosed herein. The term“machine-readable medium” includes, but not be limited to, datarepositories in the form of solid-state memories, optical media, andmagnetic media.

Additional Considerations

The moving vehicle analysis system as disclosed provides benefits andadvantages that include the transformation of clusters of pixels into adigital representation of moving vehicles, and for each moving vehicle,the digital representation of the size of the moving vehicle, the speedof the moving vehicle, and the direction of the moving vehicle. Otheradvantages of the system include faster processing of the aerial images,less power consumption, lower latency in moving vehicle detection, lessdata transmitted over the network, etc.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms, for example, as illustrated anddescribed with FIGS. 1, 2, 4, 6, 7, and 13. Modules may constituteeither software modules (e.g., code embodied on a machine-readablemedium) or hardware modules. A hardware module is tangible unit capableof performing certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module mayinclude dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also includeprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors, e.g.,processor 1302, that are temporarily configured (e.g., by software) orpermanently configured to perform the relevant operations. Whethertemporarily or permanently configured, such processors may constituteprocessor-implemented modules that operate to perform one or moreoperations or functions. The modules referred to herein may, in someexample embodiments, include processor-implemented modules.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., application program interfaces (APIs).)

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of this specification are presented in terms of algorithmsor symbolic representations of operations on data stored as bits orbinary digital signals within a machine memory (e.g., a computermemory). These algorithms or symbolic representations are examples oftechniques used by those of ordinary skill in the data processing artsto convey the substance of their work to others skilled in the art. Asused herein, an “algorithm” is a self-consistent sequence of operationsor similar processing leading to a desired result. In this context,algorithms and operations involve physical manipulation of physicalquantities. Typically, but not necessarily, such quantities may take theform of electrical, magnetic, or optical signals capable of beingstored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that includes a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the claimed invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for identifying moving vehicles from low resolutionimagery through the disclosed principles herein. Thus, while particularembodiments and applications have been illustrated and described, it isto be understood that the disclosed embodiments are not limited to theprecise construction and components disclosed herein. Variousmodifications, changes and variations, which will be apparent to thoseskilled in the art, may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope defined in the appended claims.

What is claimed is:
 1. A method, comprising: accessing a plurality ofimages of a substantially same geographical area, each image of theplurality of images captured at a separate time from each other using aseparate imaging sensor, each imaging sensor capturing informationcorresponding to a different spectral band; identifying, for theplurality of images, a set of blobs, each blob comprising a plurality ofadjacent pixels, wherein each image of the plurality of images includesa blob of the set of blobs, and wherein each blob in the set of blobshas a location in each image that differs from a location of other blobsin the set of blobs; generating a score indicating a likelihood that theset of blobs correspond to a moving object; and storing an indicationthat the set of blobs correspond to the moving object responsive to thegenerated score exceeding a threshold.
 2. The method of claim 1, whereina blob in each image is identified by: determining candidate pixellocations of each image that have pixel attribute values that are atleast a threshold difference from pixel attribute values of the samepixel locations in each of the other images in the plurality of images;and identifying pixels of the candidate pixel locations that areadjacent to each other as the blob.
 3. The method of claim 1, wherein ablob in each image is identified by: merging adjacent regions of pixelsin the image having pixels with pixel attribute values that have alowest difference amongst adjacent regions of pixels in the image; andidentifying a merged adjacent region of pixels as a blob.
 4. The methodof claim 1, wherein a blob in each image is identified by: applying asliding window to each image, the sliding window being a rectangularpixel region which is moved across the image; and identifying a blob ata location of the sliding window responsive to the sliding window beingmoved to the location of the image which has an average pixel attributevalue of pixels bounded by the sliding window that is at least athreshold difference from average pixel attribute values of pixels atthe same location in other images of the plurality of images.
 5. Themethod of claim 1, wherein each imaging sensor captures light in atleast one of a red visible light spectral band, green visible lightspectral band, and blue visible light spectral band.
 6. The method ofclaim 5, wherein the image sensor that captures light in the red visiblelight spectral captures one of the plurality of images at a first time,wherein the image sensor that captures light in the green visible lightspectral captures one of the plurality of images at a second time,wherein the image sensor that captures light in the blue visible lightspectral captures one of the plurality of images at a third time, andwherein a first interval of time between the first time and the secondtime differs from a second interval of time between the second time andthe third time.
 7. The method of claim 1, further comprising:determining a direction of the moving object based on a differencebetween the location of each blob with other blobs of the set of blobs.8. The method of claim 1, further comprising: detecting paths in theplurality of images using edge detection, the paths being regions withinthe plurality of images; and wherein identifying the set of blobsfurther comprises identifying each blob responsive to the location ofeach blob being within a region of each image detected as a path.
 9. Asystem, comprising: an aerial imaging device with a plurality of imagesensors, each image sensor of the plurality of image sensors capturingan image of a plurality of images at a separate time from each other,each imaging sensor capturing information corresponding to a differentspectral band, the plurality of images of a substantially samegeographical area; an analysis system configured to: access theplurality of images of a same geographical area; identify, for theplurality of images, a set of blobs, each blob comprising a plurality ofadjacent pixels, wherein each image of the plurality of images includesa blob of the set of blobs, and wherein each blob in the set of blobshas a location in each image that differs from a location of other blobsin the set of blobs; generate a score indicating a likelihood that theset of blobs correspond to a moving object; and store an indication thatthe set of blobs correspond to the moving object responsive to thegenerated score exceeding a threshold.
 10. The system of claim 9,wherein a blob in each image is identified by: determining candidatepixel locations of each image that have pixel attribute values that areat least a threshold difference from pixel attribute values of the samepixel locations in each of the other images in the plurality of images;and identifying pixels of the candidate pixel locations that areadjacent to each other as the blob.
 12. The system of claim 9, wherein ablob in each image is identified by: merging adjacent regions of pixelsin the image having pixels with pixel attribute values that have alowest difference amongst adjacent regions of pixels in the image; andidentifying a merged adjacent region of pixels as a blob.
 13. The systemof claim 9, wherein a blob in each image is identified by: applying asliding window to each image, the sliding window being a rectangularpixel region which is moved across the image; and identifying a blob ata location of the sliding window responsive to the sliding window beingmoved to the location of the image which has an average pixel attributevalue of pixels bounded by the sliding window that is at least athreshold difference from average pixel attribute values of pixels atthe same location in other images of the plurality of images.
 14. Thesystem of claim 9, wherein each imaging sensor captures light in atleast one of a red visible light spectral band, green visible lightspectral band, and blue visible light spectral band.
 15. The system ofclaim 9, wherein the analysis system is further configured to: determinea direction of the moving object based on a difference between thelocation of each blob with other blobs of the set of blobs.
 16. Thesystem of claim 9, wherein the analysis system is further configured to:detect paths in the plurality of images using edge detection, the pathsbeing regions within the plurality of images; and identify each blob ofthe set of blobs responsive to the location of each blob being within aregion of each image detected as a path.
 17. A non-transitory computerreadable storage medium, comprising computer-readable instructions thatwhen executed by a processor causes the processor to: access a pluralityof images of a substantially same geographical area, each image of theplurality of images captured at a separate time from each other using aseparate imaging sensor, each imaging sensor capturing informationcorresponding to a different spectral band; identify, for the pluralityof images, a set of blobs, each blob comprising a plurality of adjacentpixels, wherein each image of the plurality of images includes a blob ofthe set of blobs, and wherein each blob in the set of blobs has alocation in each image that differs from a location of other blobs inthe set of blobs; generate a score indicating a likelihood that the setof blobs correspond to a moving object; and store an indication that theset of blobs correspond to the moving object responsive to the generatedscore exceeding a threshold.
 18. The non-transitory computer readablestorage medium of claim 17, wherein a blob in each image is identifiedby: determining candidate pixel locations of each image that have pixelattribute values that are at least a threshold difference from pixelattribute values of the same pixel locations in each of the other imagesin the plurality of images; and identifying pixels of the candidatepixel locations that are adjacent to each other as the blob.
 19. Thenon-transitory computer readable storage medium of claim 17, wherein ablob in each image is identified by: merging adjacent regions of pixelsin the image having pixels with pixel attribute values that have alowest difference amongst adjacent regions of pixels in the image; andidentifying a merged adjacent region of pixels as a blob.
 20. Thenon-transitory computer readable storage medium of claim 17, wherein ablob in each image is identified by: applying a sliding window to eachimage, the sliding window being a rectangular pixel region which ismoved across the image; and identifying a blob at a location of thesliding window responsive to the sliding window being moved to thelocation of the image which has an average pixel attribute value ofpixels bounded by the sliding window that is at least a thresholddifference from average pixel attribute values of pixels at the samelocation in other images of the plurality of images.